Skip to main content

Incident Report: Low and Critical Balance Alerts on Various Chains

Date: 2023-12-02
Time: 18:53 (GMT+3)
Duration: 30 minutes

Description​

Low and critical balance alerts were reported on multiple networks. These alerts were initially thought to be unrelated to a previous issue escalated by Andrew.

Root Cause​

The alerts were primarily caused by recent changes in gas prices, which led to an underestimation of required thresholds. The updates in the dAPI prices and the subsequent need to adjust the minSponsorWalletBalances in metadata.json triggered the alerts.

Impact​

The incident resulted in several low and critical balance alerts across different networks, causing a need for immediate attention and action to resolve the discrepancies.

Timeline​

  • 18:45 - Execution of manager-multisig transactions to fund Nodary MFDs.
  • 18:53 - Initial reports of low and critical balance alerts on multiple networks.
  • 18:54 - Acknowledgment of the alerts and decision to ignore them temporarily.
  • 19:19 - Actions taken to redeploy Airseeker.
  • 19:28 - Confirmation that no managed feeds were running on certain networks, leading to a decision to remove these from the alert system.

Lessons Learned​

This incident highlighted the need for proactive monitoring and quick response to threshold changes due to fluctuating market conditions.

Actions Taken​

  • Temporary acknowledgment of false alerts.
  • Execution of transactions to withdraw and redeploy funds for Nodary MFDs.
  • Redeployment of critical infrastructure components such as wallet-watcher and merkle funder worker.
  • Manual adjustments to the configuration of various systems to align with the new thresholds.

Incident Reviewer(s)​

  • Arda, Burak, Prenaam D, Santiago, Bedirhan, Mertcan Karik, Jiri Herzan, Ugur.
Last updated on by Warren Anderson